Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Precompiled header for modules & Refine by PImpl to accelerate release build #5047

Merged
merged 18 commits into from
Jun 8, 2022

Conversation

solotzg
Copy link
Contributor

@solotzg solotzg commented Jun 2, 2022

What problem does this PR solve?

Issue Number: ref #4909

Problem Summary:

Ref: https://solotzg.github.io/2022/06/01/system/cpp-compile-optimize/#%E9%A2%84%E7%BC%96%E8%AF%91%E5%A4%B4%E6%96%87%E4%BB%B6%EF%BC%88PCH%EF%BC%89

Before

**** Time summary:
Compilation (5485 times):
  Parsing (frontend):         7181.8 s
  Codegen & opts (backend):   6224.5 s

*** Expensive headers:
1091837 ms: /root/work/tiflash/dbms/src/Common/FmtUtils.h (included 750 times, avg 1455 ms), included via:
1022467 ms: /root/work/tiflash/libs/libcommon/include/common/StringRef.h (included 754 times, avg 1356 ms), included via:
985876 ms: /root/work/tiflash/dbms/src/Common/Exception.h (included 748 times, avg 1318 ms), included via:
974680 ms: /root/work/tiflash/dbms/src/Common/StackTrace.h (included 749 times, avg 1301 ms), included via:
681167 ms: /data2/tiflash-env/sysroot/include/c++/v1/string (included 1903 times, avg 357 ms), included via:
501559 ms: /data2/tiflash-env/sysroot/include/c++/v1/algorithm (included 1969 times, avg 254 ms), included via:
498058 ms: /root/work/tiflash/dbms/src/Columns/IColumn.h (included 516 times, avg 965 ms), included via:
422329 ms: /root/work/tiflash/dbms/src/Core/Types.h (included 706 times, avg 598 ms), included via:
405688 ms: /data2/tiflash-env/sysroot/include/c++/v1/functional (included 1971 times, avg 205 ms), included via:

After

**** Time summary:
Compilation (5488 times):
  Parsing (frontend):         4208.2 s
  Codegen & opts (backend):   6663.9 s

*** Expensive headers:
188325 ms: /data2/work/tiflash/dbms/src/Columns/IColumn.h (included 515 times, avg 365 ms), included via:
...
177991 ms: /data2/work/tiflash/dbms/src/Common/Decimal.h (included 657 times, avg 270 ms), included via:
...
122434 ms: /data2/work/tiflash/dbms/src/Core/Block.h (included 415 times, avg 295 ms), included via:
...
115633 ms: /data2/work/tiflash/dbms/src/Core/Field.h (included 656 times, avg 176 ms), included via:
...

With the help of LTO(#4890, #4924), Forward declaration won't hurt performance because linker(lld) will make short functions inline automatically.

For example: use objdump -d, there will be only one entry to function PlusDecimalInferer::infer in code section. There is no other function call towards PlusDecimalInferer::infer by instruction callq.

➜  release-centos7-llvm git:(pch) ✗ ./tiflash/tiflash version
TiFlash
Release Version: v6.2.0-alpha-30-gf2a406a
Edition:         Community
Git Commit Hash: f2a406aa93f06e98e1868597f0ff25ae87ed40c5
Git Branch:      pch
UTC Build Time:  2022-06-05 07:30:43
Enable Features: jemalloc avx avx512 unwind thinlto
Profile:         RELWITHDEBINFO

Raft Proxy
Git Commit Hash:   ca2f51f94e55bdd23749dcc02ab4afb94eeb5ae5
Git Commit Branch: HEAD
UTC Build Time:    2022-06-05 07:33:25
Rust Version:      rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
Storage Engine:    tiflash
Prometheus Prefix: tiflash_proxy_
Profile:           release

➜  release-centos7-llvm git:(pch) ✗ objdump -d ./tiflash/tiflash | grep 'PlusDecimalInferer'
0000000001ce43c0 <_ZN2DB18PlusDecimalInferer5inferEjjjj>:```

➜  release-centos7-llvm git:(pch) ✗ objdump -d ./tiflash/tiflash | grep '1ce43c0'           
0000000001ce43c0 <_ZN2DB18PlusDecimalInferer5inferEjjjj>:
 1ce43c0:       55                      push   %rbp

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jun 2, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • SchrodingerZhu
  • fuzhe1989

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 2, 2022
@solotzg solotzg added type/enhancement The issue or PR belongs to an enhancement. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 2, 2022
@solotzg solotzg requested review from fuzhe1989, zanmato1984 and SchrodingerZhu and removed request for zanmato1984, fuzhe1989 and SchrodingerZhu June 2, 2022 03:17
@solotzg
Copy link
Contributor Author

solotzg commented Jun 2, 2022

/hold

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 2, 2022
@solotzg solotzg closed this Jun 2, 2022
@solotzg solotzg reopened this Jun 2, 2022
@ti-chi-bot ti-chi-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 2, 2022
@solotzg
Copy link
Contributor Author

solotzg commented Jun 2, 2022

/run-unit-test

@solotzg
Copy link
Contributor Author

solotzg commented Jun 2, 2022

/run-integration-test

@solotzg solotzg requested a review from fuzhe1989 June 6, 2022 12:56
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 7, 2022
@solotzg
Copy link
Contributor Author

solotzg commented Jun 7, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Jun 7, 2022

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FmtUtils.h                                        17                 2    88.24%          11                 2    81.82%          39                 2    94.87%           4                 0   100.00%
Common/TiFlashException.cpp                              98                98     0.00%          13                13     0.00%          95                95     0.00%          34                34     0.00%
Common/TiFlashException.h                                 8                 4    50.00%           6                 4    33.33%           6                 4    33.33%           4                 2    50.00%
DataStreams/TiRemoteBlockInputStream.h                  117               117     0.00%          17                17     0.00%         164               164     0.00%          60                60     0.00%
Flash/Coprocessor/CoprocessorReader.h                    49                49     0.00%          11                11     0.00%          80                80     0.00%          32                32     0.00%
Flash/Coprocessor/DAGStorageInterpreter.cpp             424               424     0.00%          32                32     0.00%         870               870     0.00%         282               282     0.00%
Flash/Coprocessor/GenSchemaAndColumn.cpp                 10                 2    80.00%           2                 0   100.00%          32                 6    81.25%          10                 3    70.00%
Flash/Coprocessor/RemoteRequest.cpp                      31                31     0.00%           3                 3     0.00%          80                80     0.00%          22                22     0.00%
Flash/Coprocessor/RemoteRequest.h                         1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Flash/Coprocessor/TablesRegionsInfo.cpp                  41                41     0.00%           5                 5     0.00%          62                62     0.00%          26                26     0.00%
Flash/Coprocessor/TablesRegionsInfo.h                    14                13     7.14%           8                 7    12.50%          27                26     3.70%           6                 6     0.00%
Flash/Management/tests/gtest_manual_compact.cpp         905               315    65.19%          16                 0   100.00%         251                 4    98.41%         264               141    46.59%
Flash/Mpp/MinTSOScheduler.h                               4                 1    75.00%           2                 0   100.00%           4                 0   100.00%           4                 3    25.00%
Server/CLIService.h                                      37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Storages/Transaction/TMTContext.h                         3                 2    33.33%           3                 2    33.33%           3                 2    33.33%           0                 0         -
TestUtils/ColumnsToTiPBExpr.cpp                          55                 5    90.91%           6                 0   100.00%         122                11    90.98%          40                 7    82.50%
TiDB/Schema/TiDBSchemaSyncer.h                          140               132     5.71%          13                 9    30.77%         125               100    20.00%          52                51     1.92%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  1954              1274    34.80%         156               113    27.56%        2070              1616    21.93%         864               693    19.79%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18284      9733             46.77%    205116  97578        52.43%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 7, 2022
@solotzg
Copy link
Contributor Author

solotzg commented Jun 7, 2022

/merge

@ti-chi-bot
Copy link
Member

@solotzg: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 5662fdd

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2022
@solotzg
Copy link
Contributor Author

solotzg commented Jun 7, 2022

/hold

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 7, 2022
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Jun 7, 2022

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FmtUtils.h                                        17                 2    88.24%          11                 2    81.82%          39                 2    94.87%           4                 0   100.00%
Common/TiFlashException.cpp                              98                98     0.00%          13                13     0.00%          95                95     0.00%          34                34     0.00%
Common/TiFlashException.h                                 8                 4    50.00%           6                 4    33.33%           6                 4    33.33%           4                 2    50.00%
DataStreams/TiRemoteBlockInputStream.h                  117               117     0.00%          17                17     0.00%         164               164     0.00%          60                60     0.00%
Flash/Coprocessor/CoprocessorReader.h                    49                49     0.00%          11                11     0.00%          80                80     0.00%          32                32     0.00%
Flash/Coprocessor/DAGStorageInterpreter.cpp             424               424     0.00%          32                32     0.00%         870               870     0.00%         282               282     0.00%
Flash/Coprocessor/GenSchemaAndColumn.cpp                 10                 2    80.00%           2                 0   100.00%          32                 6    81.25%          10                 3    70.00%
Flash/Coprocessor/RemoteRequest.cpp                      31                31     0.00%           3                 3     0.00%          80                80     0.00%          22                22     0.00%
Flash/Coprocessor/RemoteRequest.h                         1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Flash/Coprocessor/TablesRegionsInfo.cpp                  41                41     0.00%           5                 5     0.00%          62                62     0.00%          26                26     0.00%
Flash/Coprocessor/TablesRegionsInfo.h                    14                13     7.14%           8                 7    12.50%          27                26     3.70%           6                 6     0.00%
Flash/Management/tests/gtest_manual_compact.cpp         905               315    65.19%          16                 0   100.00%         251                 4    98.41%         264               141    46.59%
Flash/Mpp/MinTSOScheduler.h                               4                 1    75.00%           2                 0   100.00%           4                 0   100.00%           4                 3    25.00%
Interpreters/Context.h                                   13                 5    61.54%          13                 5    61.54%          13                 5    61.54%           0                 0         -
Server/CLIService.h                                      37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Storages/Transaction/KVStore.cpp                        387                76    80.36%          41                 3    92.68%         675                79    88.30%         208                75    63.94%
Storages/Transaction/KVStore.h                            5                 3    40.00%           5                 3    40.00%           5                 3    40.00%           0                 0         -
Storages/Transaction/ProxyFFI.cpp                       174               110    36.78%          41                24    41.46%         346               235    32.08%          68                31    54.41%
Storages/Transaction/ReadIndexWorker.cpp                299                 8    97.32%          80                 3    96.25%         693                 9    98.70%         170                10    94.12%
Storages/Transaction/TMTContext.h                         3                 2    33.33%           3                 2    33.33%           3                 2    33.33%           0                 0         -
TestUtils/ColumnsToTiPBExpr.cpp                          55                 5    90.91%           6                 0   100.00%         122                11    90.98%          40                 7    82.50%
TiDB/Schema/TiDBSchemaSyncer.h                          140               132     5.71%          13                 9    30.77%         125               100    20.00%          52                51     1.92%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  2832              1476    47.88%         336               151    55.06%        3802              1947    48.79%        1310               809    38.24%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18284      9733             46.77%    205113  97577        52.43%

full coverage report (for internal network access only)

@solotzg
Copy link
Contributor Author

solotzg commented Jun 7, 2022

/merge

@ti-chi-bot
Copy link
Member

@solotzg: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 96567cd

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Jun 7, 2022

Coverage for changed files

Filename                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FmtUtils.h                                        17                 2    88.24%          11                 2    81.82%          39                 2    94.87%           4                 0   100.00%
Common/TiFlashException.cpp                              98                98     0.00%          13                13     0.00%          95                95     0.00%          34                34     0.00%
Common/TiFlashException.h                                 8                 4    50.00%           6                 4    33.33%           6                 4    33.33%           4                 2    50.00%
DataStreams/TiRemoteBlockInputStream.h                  117               117     0.00%          17                17     0.00%         164               164     0.00%          60                60     0.00%
Flash/Coprocessor/CoprocessorReader.h                    49                49     0.00%          11                11     0.00%          80                80     0.00%          32                32     0.00%
Flash/Coprocessor/DAGStorageInterpreter.cpp             424               424     0.00%          32                32     0.00%         870               870     0.00%         282               282     0.00%
Flash/Coprocessor/GenSchemaAndColumn.cpp                 10                 2    80.00%           2                 0   100.00%          32                 6    81.25%          10                 3    70.00%
Flash/Coprocessor/RemoteRequest.cpp                      31                31     0.00%           3                 3     0.00%          80                80     0.00%          22                22     0.00%
Flash/Coprocessor/RemoteRequest.h                         1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Flash/Coprocessor/TablesRegionsInfo.cpp                  41                41     0.00%           5                 5     0.00%          62                62     0.00%          26                26     0.00%
Flash/Coprocessor/TablesRegionsInfo.h                    14                13     7.14%           8                 7    12.50%          27                26     3.70%           6                 6     0.00%
Flash/Management/tests/gtest_manual_compact.cpp         905               315    65.19%          16                 0   100.00%         251                 4    98.41%         264               141    46.59%
Flash/Mpp/MinTSOScheduler.h                               4                 1    75.00%           2                 0   100.00%           4                 0   100.00%           4                 3    25.00%
Interpreters/Context.h                                   13                 5    61.54%          13                 5    61.54%          13                 5    61.54%           0                 0         -
Server/CLIService.h                                      37                37     0.00%           7                 7     0.00%         109               109     0.00%          24                24     0.00%
Storages/Transaction/KVStore.cpp                        387                77    80.10%          41                 3    92.68%         675                79    88.30%         208                77    62.98%
Storages/Transaction/KVStore.h                            5                 3    40.00%           5                 3    40.00%           5                 3    40.00%           0                 0         -
Storages/Transaction/ProxyFFI.cpp                       174               110    36.78%          41                24    41.46%         346               235    32.08%          68                31    54.41%
Storages/Transaction/ReadIndexWorker.cpp                299                 8    97.32%          80                 3    96.25%         693                 9    98.70%         170                11    93.53%
Storages/Transaction/TMTContext.h                         3                 2    33.33%           3                 2    33.33%           3                 2    33.33%           0                 0         -
TestUtils/ColumnsToTiPBExpr.cpp                          55                 5    90.91%           6                 0   100.00%         122                11    90.98%          40                 7    82.50%
TiDB/Schema/TiDBSchemaSyncer.h                          140               132     5.71%          13                 9    30.77%         125               100    20.00%          52                51     1.92%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                  2832              1477    47.85%         336               151    55.06%        3802              1947    48.79%        1310               812    38.02%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18284      9733             46.77%    205113  97578        52.43%

full coverage report (for internal network access only)

@solotzg
Copy link
Contributor Author

solotzg commented Jun 8, 2022

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2022
@ti-chi-bot ti-chi-bot merged commit 5847f1c into pingcap:master Jun 8, 2022
@solotzg solotzg deleted the pch branch June 8, 2022 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants